Skip to content

Expose configs via pipeline element table - #162

Merged
jogrogan merged 2 commits into
mainfrom
jogrogan/pipelineConfigs
Sep 10, 2025
Merged

jogrogan merged 2 commits into
mainfrom
jogrogan/pipelineConfigs

Conversation

@jogrogan

@jogrogan jogrogan commented Sep 5, 2025

Copy link
Copy Markdown
Collaborator

Expose configs via pipeline element table

0: Hoptimator> select NAME, CONFIG from "k8s".pipeline_elements;
WARN Object default/TableTrigger/ads-database-pageviews-trigger considered ready by default.
INFO TableTrigger/ads-database-pageviews-trigger is ready.
WARN Object default/FlinkSessionJob/ads-database-audience considered ready by default.
INFO FlinkSessionJob/ads-database-audience is ready.
WARN Object default/SqlJob/test-sqljob considered ready by default.
INFO SqlJob/test-sqljob is ready.
+---------------------------------------------+-------------------------------------------------------------+
|                    NAME                     |                            CONFIG                           |
+---------------------------------------------+-------------------------------------------------------------+
| SqlJob/test-sqljob                          | {"pipeline":"jogrogan-test","key1":"value1","key2":"value2"}|
| FlinkSessionJob/ads-database-audience       | {}                                                          |
| TableTrigger/ads-database-pageviews-trigger | {}                                                          |
+---------------------------------------------+-------------------------------------------------------------+
3 rows selected (0.085 seconds)

Configs should never contain sensitive information or they will be queryable

@jogrogan
jogrogan requested a review from ryannedolan September 9, 2025 14:24
@ryannedolan

Copy link
Copy Markdown
Collaborator

I wonder why we wouldn't include the whole spec, instead of jut the configs?

@jogrogan

Copy link
Copy Markdown
Collaborator Author

I wonder why we wouldn't include the whole spec, instead of jut the configs?

Configs just seems more general. We could include the whole spec but it'll be dependent on whatever downstream workflows to parse the spec correctly by type

@jogrogan
jogrogan merged commit af27482 into main Sep 10, 2025
1 check passed
@jogrogan
jogrogan deleted the jogrogan/pipelineConfigs branch September 10, 2025 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants